Server freezes (Fixed!)

  1. ‹ Older
  2. 8 years ago

    Might the issue be cause by a single module in ACE and ALiVE conflicting?
    I put up my Operation Landlord with ACE today, and it's been running about 12 hours with no crash/hang.

    I had 4 players on earlier an nothing happened. We used Arty, called Supplies, used transport and CAS helis. Ran around in a stolen T100 and killed a bunch of CSAT.

    I am still using V16 for the Perf EXE as I haven't been able to get it updated to the latest binary yet.
    I will get you a RPT if needed.

  3. Edited 8 years ago by JerkinMerkin

    Hi guys.

    I have tested with 1.54 running only @ALiVE, @CBA, and @AliveServer and my server froze around 7 hours 30min. With @ace the server froze anywhere from 10min to 1 hour. Therefore it appears to me that @ace is not the issue, it just makes the issue worse.

    More info can be found here:

  4. Edited 8 years ago by uk_apollo

    First I'm going to say what a wonderful mod @alive is, and how it has been enjoyed by all of us at 3 Commando Brigade on every single one of our busy public server missions for well over a year. Awesome job, thanks!

    Unfortunately we have also been suffering with the "freezes" in this winter weather. Identical to all the reports you've previously had here. We do run a large modset so I won't bore you with rpt's as I know you need minimal mod installations to help find the solution. But I thought it worth reporting the issue to add to the data-set, especially as we have built up a lot of experience running Alive on 24/7 dedi servers catering for 20-40 players on most days of the year.

    Symptoms

    • players see others freeze in place or walking on the spot
    • vehicles rubber band
    • players will claim to be at the same place but not see each other
    • players may not be able to join in progress
    • red chains sometimes, but not always
    • desync = 0 in map/player info
    • suddenly occurs for all players, seemingly randomly from 5 minutes to 3 hours into a mission
    • server rpt huge, spammed with specific error messages (example below)
    • server reports very low on virtual memory (memory leak)

    Facts

    • it started happening sometime after the A3 v1.54 update and/or updates to @alive and @ace3 at that time. The same modset and same missions prior to that were stable.
    • the problem goes away if you remove @alive from a larger modset
    • it's still freezing with the Eden update
    • not all of our missions freeze: we have a number which use @alive/@ace3/@rhs_ amongst other mods, which appear to be 100% stable (10 hours+ up-time)

    Our conclusions so far

    • it's not related to spawn density (happens with minimal ai numbers spawned, even with all players still at base)
    • it's not player number related (has happened with just 2)
    • it's not related specifically to being in combat (has happened with all players in a blacklisted base)
    • it's probably not a specific @ace3 module setting, as all of our missions have identical parameters (ballistics etc); some work, others don't
    • it's probably not the 3rd party Opfor type selected in the @alive modules - server freezes with BIS AAF enemy too (although to be fair, RHS/Leights is still loaded in the modset, but not being used)
    • it doesn't appear to be related to a particular @alive module (such as C2Istar); I knocked it back to the bare minimum last night and still saw rpt spam; asymmetric or conventional warfare alike
    • it probably is a conflict between @alive and either @ace3 or @rhs_ (possibly @leights_opfor) apparent only since an update from about a month ago, in agreement with the consensus opinion here

    Server RPT sample
    17:39:30 WARNING: SuppressCache::Trace called with huge input. Function exited to prevent program stall inside. tBeg[-0.000000,165.000000,-0.818642]; tEnd[0.000000,0.574304,12276.468750]; ammoHit(1.745155); distance(0.000000)
    17:39:30 In last 500 miliseconds was lost another 2 these messages.
    17:39:49 Virtual memory total 4095 MB (4294836224 B)
    17:39:49 Virtual memory free 57 MB (59785216 B)
    17:39:49 Physical memory free 25890 MB (27147853824 B)
    17:39:49 Page file free 56668 MB (59421413376 B)
    17:39:49 Process working set 3530 MB (3701755904 B)
    17:39:49 Process page file used 3762 MB (3944747008 B)
    17:39:49 Longest free VM region: 3604480 B
    17:39:49 VM busy 4233289728 B (reserved 150560768 B, committed 4082728960 B, mapped 74436608 B), free 61546496 B
    17:39:49 Small mapped regions: 24, size 106496 B
    17:41:40 NetServer::finishDestroyPlayer(1970388062): DESTROY immediately after CREATE, both cancelled

    The WARNING message is spammed, thousands of times.

    www.3commandobrigade.com

  5. ditto to UK apollo. Ran my British Steel mission with another friend for over 2 hours with no issues. Thought we were in the clear. Called for CAS and a Helo Transport. As soon as we are about to land on the helo....desync.

    Checked the RPT...same out of memory crash that apollo mentioned above.

    Immediately restarted the server and launched the helicopters again, told them to land...landed fine this time.

    So....in conclusion....still getting crashes....not happening "consistently" in any one instance.

    My British Steel mission uses ALiVE + ACE...no RHS. 3CB and CAF Aggressors are the unit packs.

    I will say...there is a definite difference though in performance. I think we're close.

  6. Edited 8 years ago by uk_apollo

    A3 1.56
    @cba_a3 v2.3.0
    Yes to cqb module.

    One of our public server Alive missions repeatedly fails with freezes so we stopped running it for now.

    With the same modset (incl. @alive, @ace3, @rhs_afrf3) another Alive mission has worked fine all day and was up for 28 hours.

    https://www.gametracker.com/server_info/144.76.162.200:2362/
    Around 25-30 players this evening.

    I don't envy you folks trying to sort this problem out!

  7. Tupolov

    Show in context Administrator

    @uk_apollo A3 1.56
    @cba_a3 v2.3.0
    Yes to cqb module.

    One of our public server Alive missions repeatedly fails with freezes so we stopped running it for now.

    With the same modset (incl. @alive, @ace3, @rhs_afrf3) another Alive mission has worked fine all day and was up for 28 hours.

    https://www.gametracker.com/server_info/144.76.162.200:2362/
    Around 25-30 players this evening.

    I don't envy you folks trying to sort this problem out!

    can u share both missions?

  8. I guess i can piggy-back on this thread regarding the server hangs and crashes.

    My friend and I tried to make an Asymetric mission the other week and started to have these errors, mostly the server hangs and crashes after its memory has been filled up.

    So we dug into the problem and started a dev with @ALiVE (1.0.0.1602091) and @CBA (2.3.1) only (using the stable branch). The mission is made on Altis and in the "Eden" editor the mission.sqm binarization is turned off. Here are my impressions:

    -It seems to come from the Installation placement in the MIlitary AI commander, when activated, in local server i get the following error "Error undefined variable in expression: _state" "x\alive\addons\fnc_analysis\fnc_liveanalysis.sqf, line 514" when you start running trough the map towards the enemy "bastion" once the modules loaded up (until the Intel appears).

    -Removing the Assymetric mode or just simply deactivating the Installations stops the error. Other modes like Invasion and Occupation doesnt generate the error AFAIK.

    When loaded up on the dedicated, you can start the mission, load it up at 100%, leave it run for as many hours/days as you want it wont crash, its only when you log in and start walking towards the enemy that the error occurs. It freezes the server and chugs up all the RAM until the dedicated server crashes. You won't notice this if you play alone, only that the server on "#monitor" doesn't post anymore after a while.

    I did a "portfolio" with the mission file, and the two RPT's (one from my local machine testing in local host, and the other from our dedicated (*HUGE RPT 24mb*)) and also a screencapture of the error i got in local host running this mission. The mission inside is quite light (couple of kbs) but its made to stress test our machine also, we didnt want to overlook certain aspects, so if im correct this mission takes a bit to load up and has 86 objectives triggering the "80" limit warning. Also, if the mission is badly "structured" with the custom objectives let me know, for all i know maybe im trying to do something impossible with ALiVE and just plainly breaking it because im overwhelming the Military Commander with multiple Custom Objectives (is there a limit to the number you can synch to it?)

    Ive read that it might be the error that HighHead is working on but i might be mistaken. Hoping for the best.

    FTP download

    Hope this helps

  9. Ok, good to hear. With the current build, I had the impression - as already mentioned in the previous thread by numerous people - that no single mod alone causes the freezes, but that each additional mod on top of the (by the functions you identified as per above posting) already strained server can make him more and more unstable up until the point where he crashes.

    As a temporary fix I tested to play either with RHS OR with ACE, but not with both. This increased my server stabilty to a level where playing for 2-5 hours was possible. But mostly I was alone on my server and only once played for about 3 hours with a buddy in a No-ACE-COIN Mission.

    Happy to hear that a fix is on the way and will gladly test the new build with the same modset I use now.

  10. I've stress tested the hell out of the new test build and British Steel ran for almost 40 hours straight with 10 people in and out of mission....no freezes. Well done Tup and Co.

  11. As Jooce said, SOAR has tested this update twice. We ran a 50 min initial test run, everything went smooth. We then learned that since the Eden update came out, Eden missions and headless client don't work well together. So I created a 2D mission and got HC and Alive working. That mission has now been up for 3 hours with no issues. We will be adding it into our groups mod pack and doing large scale testing this weekend. Thanks Alive team!

  12. @SpyderBlack723 Fix deployed in latest release!

    https://forums.bistudio.com/topic/187954-alive-advanced-light-infantry-virtual-environment-10-ga/page-11#entry2996372

    Man that was fast! Will try it out in a couple of hours, thank you so much